Skip to content

fix(temporal): guard int64 overflow in DATE/TIMESTAMP decomposition - #386

Merged
singaraiona merged 3 commits into
RayforceDB:devfrom
belowzeroff:fix/temporal-extract-overflow
Aug 9, 2026
Merged

fix(temporal): guard int64 overflow in DATE/TIMESTAMP decomposition#386
singaraiona merged 3 commits into
RayforceDB:devfrom
belowzeroff:fix/temporal-extract-overflow

Conversation

@belowzeroff

Copy link
Copy Markdown
Contributor

Problem

The DATE/TIMESTAMP → microseconds conversion shared by the temporal extract and truncate paths overflowed int64 on extreme inputs (UBSan):

(dd (as 'DATE 2147483647))
src/ops/temporal.c:105: runtime error: signed integer overflow:
  2147483647 * 86400000000 cannot be represented in type 'long long'

(mm (as 'TIMESTAMP -9223372036854775807))
src/ops/temporal.c:109: runtime error: signed integer overflow:
  9223372036854775807 + 999 cannot be represented in type 'long long'
  • DATE is int32 days, so an extreme value × µs-per-day overflows.
  • The TIMESTAMP ns→µs floor negated the input (-((-raw)+999)/1000), so a value within 999 of INT64_MIN overflowed.

Both appear in all four decomposition kernels: the standalone ray_temporal_extract / ray_temporal_truncate and the DAG exec_extract / exec_date_trunc morsel kernels (reachable via (dd d) and via dotted-path col.yyyy / col.date in select).

Fix

  • Do the TIMESTAMP ns→µs floor with truncate-then-adjust so it never negates — overflow-free and exact at INT64_MIN.
  • A DATE so extreme its µs value is not representable (hundreds of millennia out) decodes to a null instead of reading overflow garbage, consistent with how these kernels already treat a null input.
  • Truncate additionally nulls a result whose bucketed µs would overflow the int64 nanosecond output.

Behaviour

Expression Before After
(yyyy (as 'DATE 2147483647)) UB 0Nl (null)
(mm (as 'TIMESTAMP -9223372036854775807)) UB 9
(yyyy 2024.03.15) 2024 2024 (unchanged)
select d.yyyy over [0 366 2147483647] UB [2000 2001 0Nl]
select d.date over the same UB [… … 0Np]

Ordinary values are unaffected. New coverage in temporal/extract_trunc_overflow.rfl exercises the standalone and both DAG kernels with extreme DATE, minimum-edge TIMESTAMP, and ordinary values. Full make test passes (3660/3661, 1 skipped, 0 failed) under the default ASan/UBSan build.

🤖 Generated with Claude Code

The DATE/TIMESTAMP → microseconds conversion shared by the temporal
extract and truncate paths overflowed int64 on extreme inputs (UBSan):

  * DATE is int32 days, so an extreme value × µs-per-day overflowed —
    (dd (as 'DATE 2147483647)) tripped `raw * 86400000000`.
  * The ns→µs floor for TIMESTAMP negated the input, so a value within
    999 of INT64_MIN overflowed `(-raw) + 999`.
  * The DAG date_trunc YEAR/MONTH arms re-multiply days_from_civil(...) —
    a day count floored down to the period start, up to a year beyond the
    input — by µs-per-day, so a DATE that cleared the µs bound still
    overflowed (d.year of (as 'DATE -106751991)).

Both conversions appear in all four decomposition kernels: the standalone
ray_temporal_extract / ray_temporal_truncate and the DAG exec_extract /
exec_date_trunc morsel kernels.

Do the TIMESTAMP ns→µs floor with truncate-then-adjust so it never
negates (overflow-free, exact at INT64_MIN). A DATE so extreme its µs
value is not representable decodes to a null instead of reading overflow
garbage, consistent with how these kernels already treat a null input.
The truncate DATE bound is the int64-NANOSECOND representable day range
(not just the µs one) so the YEAR/MONTH re-multiply cannot overflow, and
truncate additionally nulls a result whose bucketed µs would overflow the
ns output.

Adds temporal/extract_trunc_overflow.rfl covering the standalone kernels
(yyyy/dd/mm/hh and (date …)) and the DAG kernels (dotted col.field),
including the year/month re-multiply path, large-magnitude negative DATE,
and minimum-edge TIMESTAMP.
@belowzeroff
belowzeroff force-pushed the fix/temporal-extract-overflow branch from bfa94e1 to ae215c5 Compare August 7, 2026 11:12
@belowzeroff

Copy link
Copy Markdown
Contributor Author

Addressed the rayforce-audit findings in ae215c59.

Blocking (1) — incomplete truncate guard. Correct: the exec_date_trunc YEAR/MONTH arms re-multiply days_from_civil(...) * µs-per-day on a day count floored down to the period start (up to a year beyond the input), so a DATE that cleared the µs bound (INT64_MAX / µs-per-day) still overflowed — d.year of (as 'DATE -106751991) tripped it (-106752347 * 86400000000). Tightened the IN32 DATE guard to the int64-nanosecond representable day range (INT64_MAX / 1000 / DT_USEC_PER_DAY), matching the sibling rte_trunc_elem; the re-multiply and the final ×1000 now both stay in range.

(2) — vacuous assertion. Right, -2147483648 == NULL_I32, so it exited at the null-input early-out. Replaced with -200000000 (a real non-null negative that exercises the guard's negative branch).

(3) — untested standalone truncate / overstated header. Added (date <TIMESTAMP>) / (date <DATE>) cases (ordinary → day-truncated, out-of-ns-range → null) and the DAG d.year / d.month re-multiply path with a large-magnitude negative DATE. Corrected the header to describe exactly what is covered.

(4) — conservative low-edge guard. Kept the unconditional one-bucket headroom (avoids a branch in the hot kernel) and documented that it rounds an immaterial ~1-bucket band at the extreme low edge (~292 millennia before 2000) to null.

Full make test still green (3660/3661, 1 skipped, 0 failed) under ASan/UBSan; the d.year (as 'DATE -106751991) repro no longer reports UB.

@singaraiona singaraiona left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found one correctness issue at the lower timestamp boundary. The overflow fixes otherwise look sound, CI is green, and the full local ASan/UBSan suite passed (3664/3664).

Comment thread src/ops/temporal.c Outdated
static inline bool rte_trunc_elem(int8_t t, int64_t raw, int64_t bucket, int64_t* out_ns) {
int64_t us;
if (!rte_to_us_ck(t, raw, &us)) return false;
if (us > INT64_MAX / 1000LL || us < INT64_MIN / 1000LL + bucket) return false;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] This unconditional bucket headroom rejects valid timestamps before checking the actual truncated result. For example, (date (as 'TIMESTAMP -9223286400000000000)) returns 0Np, although that value is already the representable day boundary 1707.09.23D00:00:00.000000000. The equivalent DAG path (select ts.date) returns that timestamp, so the two public paths disagree. Please compute the bucketed value with overflow-safe arithmetic, range-check the actual result, and add this boundary as a regression.

rte_trunc_elem rejected a truncation whenever the input fell within one
bucket of the low int64-ns edge, even when the floored result was still
representable — so (date (as 'TIMESTAMP -9223286400000000000)) returned 0Np
though its day boundary 1707.09.23 is a valid TIMESTAMP that the DAG path
(select ts.date) returned, leaving the two public truncate paths disagreeing
(PR RayforceDB#386 review).

Floor with overflow-safe arithmetic (truncate toward zero, then guard the
single toward-minus-infinity bucket subtraction) and range-check the actual
bucketed result against the ns domain, mirroring exec_date_trunc. Adds the
low-boundary case — standalone and DAG — as a regression.
@belowzeroff

Copy link
Copy Markdown
Contributor Author

Addressed the review in 686abd1f.

rte_trunc_elem now floors with overflow-safe arithmetic (truncate toward zero, then a single guarded toward-−∞ bucket subtraction) and range-checks the actual bucketed result against the int64-ns domain — mirroring exec_date_trunc — instead of the unconditional pre-floor + bucket headroom.

(date (as 'TIMESTAMP -9223286400000000000)) now returns 1707.09.23D00:00:00.000000000, matching the DAG ts.date path, so the two public truncate paths agree at the boundary. The extreme-min case (-9223372036854775807, whose floored day boundary genuinely spills past the ns range) still nulls.

Added that boundary — standalone (date …) and DAG ts.date — as a regression in extract_trunc_overflow.rfl. Full make test green (3660/3661, 1 skipped, 0 failed) under ASan/UBSan; the repro no longer disagrees across paths.

@singaraiona
singaraiona merged commit 8a1cf0c into RayforceDB:dev Aug 9, 2026
9 checks passed
singaraiona added a commit that referenced this pull request Aug 11, 2026
* v2.4.0 (#327)

* feat(query): support live inserts into parted tables

Add immutable live-tail growth with explicit partition keys, shared FILE-domain symbol handling, atomic symbol rebinding, adversarial coverage, documentation, and a runnable rollover example.

* fix(core): restore total-core -c semantics

* fix(parse) Fix nonstring if not defined

* fix(store): surface FlushFileBuffers failure in journal SYNC mode (#335)

In RAY_JOURNAL_SYNC mode ray_journal_write_bytes checks fsync's return on
POSIX and fails the write with RAY_ERR_IO, but the Windows branch ignored
FlushFileBuffers' return. A failed flush there was silently swallowed, so
SYNC mode reported success while the data may not have reached disk —
dropping the durability guarantee the mode exists to provide.

Check FlushFileBuffers (0 = failure) and return RAY_ERR_IO, mirroring the
POSIX path. Windows-only branch (not built on the Linux/macOS CI matrix),
so it is verified by inspection against the adjacent fsync check; the
failure path is not unit-testable, like the existing POSIX one.

Co-authored-by: Evgen Byelozorov <belowzeroff@gmail.com>

* fix(hnsw): reject build dims whose vector count overflows size_t (#333)

ray_hnsw_build sized the copied vector block as n_nodes * dim *
sizeof(float) with no overflow check. Dimensions whose product wraps
size_t under-allocate the copy while the memcpy — and every later distance
read (vectors + id*dim) — run past the buffer. Guard the product before any
allocation, mirroring the per-layer neighbor guard in the loader, and
reject overflowing dimensions.

This hardens the public C API boundary; the in-tree (hnsw-build ...) path
sizes vectors from an in-memory list and cannot reach the overflow, so it
is defense-in-depth.

Add a regression test driving an overflowing n_nodes/dim pair; with the
guard removed it faults under ASan (stack-buffer-overflow at the copy).

Co-authored-by: Evgen Byelozorov <belowzeroff@gmail.com>
Co-authored-by: Anton Kundenko <singaraiona@gmail.com>

* fix(store): read full link sidecar to avoid wrong-symbol truncation (#334)

try_load_link_sidecar read the target table's sym name into a fixed
256-byte buffer (fread of 255 bytes). A name longer than 255 bytes was
silently truncated, so ray_sym_intern interned a DIFFERENT symbol and the
loaded column linked to the wrong table — silent data corruption on a
save/load round-trip. The writer already emits the full, untruncated name.

Read the whole sidecar into a buffer sized to the file (capped at 1 MiB to
bound a corrupt/oversized file), and reject a short read (fread returning
fewer bytes than the file size — an I/O error or a race-truncated sidecar)
so a partial name can't be interned as a different symbol either.

Add a regression test that links through a 300-byte target name and asserts
the loaded link_target matches; it fails without the fix.

Co-authored-by: Evgen Byelozorov <belowzeroff@gmail.com>
Co-authored-by: Anton Kundenko <singaraiona@gmail.com>

* fix(hnsw): reject index files whose vector count overflows size_t (#332)

hnsw_load_impl read n_nodes and dim straight from the file header and
sized the vectors allocation as n_nodes * dim * sizeof(float) with no
overflow check. A crafted header could make that product wrap size_t, so
ray_sys_alloc under-allocated the buffer while the following fread still
read the full (large) element count and wrote past the allocation — a
heap-overflow write driven by an untrusted index file.

Factor the check into ray_hnsw_vec_size_valid(n_nodes, dim) and reject the
header before any allocation, mirroring the per-layer neighbor guard.

Add a unit test that drives the helper directly (ordinary dims, non-positive
dims, an overflowing pair, and the exact size_t boundary). It is tested at
the helper rather than through ray_hnsw_load because an overflow-patched
header is refused earlier — the huge node-level read fails first — so a
full-load test could not distinguish the guard.

Co-authored-by: Evgen Byelozorov <belowzeroff@gmail.com>

* fix(docs): remediate F-0001 F-0005 F-0007

Escalate F-0002, F-0003, F-0004, and F-0006 into CF-0001 through CF-0004 after the required corpus census.

* fix(docs): remediate CF-0001

* fix(docs): remediate CF-0002

* chore(audit): plan CF-0003 ratification

* fix(docs): remediate CF-0003

* feat(docs): redesign website and documentation

Rebuild the MkDocs and marketing surfaces around the Rayforce brand, add the live market demo and cloud preview, unify responsive navigation, and eliminate reload layout shifts.

* fix(null): avoid f64 null casts to integers (#340)

* fix(null): avoid f64 null casts to integers

* fix(expr): guard f64 to i64 fallback casts

* fix(null): clamp finite f64 narrow casts

---------

Co-authored-by: Evgen Byelozorov <belowzeroff@gmail.com>

* fix(expr): avoid null truthiness casts in fallback binary ops (#339)

* fix(expr): avoid null truthiness casts in fallback binary ops

binary_range's fallback OP_AND/OP_OR kernels cast the widened `double`
operand straight to `uint8_t`:

    uint8_t li = (uint8_t)LV_READ(i);

`LV_READ` widens integer operands to `double` and yields NaN for float
nulls, so this had two defects:
  - Wrong answers from 8-bit truncation: `(uint8_t)256.0 == 0`, so
    `256 and 1b` returned false.
  - Undefined behavior: casting NaN (NULL_F64) or a widened NULL_I64
    (-9.2e18) to `uint8_t` is UB per C11 6.3.1.4.
UBSan flagged the latter via expr_null/diff_i64_{and,or}_raw and
expr_null/diff_f64_andor_chokes.

Route AND/OR through two truthiness helpers that compare on the widened
double and never cast it back to an integer:
  - truthy_intish(v, nullv) — false for 0 and for the operand's null
    sentinel. The fallback reads raw column memory, so a null arrives as
    the per-type sentinel widened to double (NULL_I16 / NULL_I32 /
    NULL_I64, with DATE/TIME stored as I32) rather than the NULL_I64 the
    VM kernel sees; `nullv` is derived per operand from the bound pointer
    type so I16/I32 nulls read as false, not just I64.
  - truthy_f64ish(v) — false for 0.0 and NaN (float null).

Non-null truthiness is unchanged and null-input positions still agree
with the VM kernel (documented "AND/OR with any null operand -> 0" and
the fix_null_comparisons post-pass), keeping fallback ≡ fused.

Add regression tests pinning fallback ≡ fused for nullable I64, I32 and
I16 AND/OR operands (expr_null/diff_i{64,32,16}_{and,or}_raw).

* fix(expr): preserve near-sentinel i64 truthiness

---------

Co-authored-by: Evgen Byelozorov <belowzeroff@gmail.com>
Co-authored-by: Anton Kundenko <singaraiona@gmail.com>

* ci: make Rayforce audit PR comments best-effort

* ci: publish Rayforce audit comments from trusted workflow

* ci: resolve fork PRs for audit commenter

* perf: parallelize serial stages around group-by; unify binary-agg null semantics (#341)

* wip(group): parallel binary aggregates (pearson/wavg/cov) via DA path

Route binary co-moment aggregators through the dense-array (DA) group path
instead of the hash scatter path.  Adds sum_y/sumsq_y/sumxy co-moment slots
to da_accum_t + per-row accumulation + per-worker merge; emit_agg_columns
already finalises PEARSON/COV/WAVG/WSUM from the co-moments.

Fixes poor multi-thread scaling of by-key binary aggregates (was ~2x, DA
path scales ~9-12x like stddev).  Verified vs numpy; diff comparator relaxed
to 1e-9 combined abs+rel (1e-12 absolute tested bit-identical summation).

Includes temporary RAY_GRPPROF phase instrumentation (to remove).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(group): merge binary-agg Sx as double for integer x-columns

wavg/pearson accumulate Sx as double even when the x column is integer
(e.g. wavg(bsize,bid), bsize=I32).  The per-worker merge dispatched on the
x-column type -> read the double bits as int64 -> garbage at >1 worker.
Force float merge for binary aggs at all 3 sum-merge sites.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(group): merge binary-agg co-moments in parallel da_merge_fn path

The parallel slot-range merge (da_merge_fn, taken when n_slots>=1024)
merged sumsq but not the binary-aggregate co-moment arrays
(sum_y/sumsq_y/sumxy).  Multi-key pearson/cov/wavg over >=1024 dense
slots produced wrong results at >1 worker.  Add the DA_NEED_PAIR merge.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore(group): remove temporary RAY_NO_V2/RAY_GRPPROF instrumentation

The binary-agg DA fix lands on the default path (v2 declines CHAR-keyed
binary group-bys -> legacy DA), so the debug env overrides are no longer
needed.  3635/3635 tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* perf(if): parallel elementwise OP_IF fill; route trivial-branch if to eager

exec_if always took the 'selected' lazy-branch path, whose scaffolding
(true-count, id-list build, per-branch gather, scatter) is serial over
ALL rows — every if-projection ran at single-core speed regardless of -c
(100M numeric if: 2.2s at any core count).

1. exec_if_eager: one shared fixed-width elementwise fill, dispatched
   across the worker pool for len >= 64K (SYM sides warm their runtime-id
   LUT serially first — sym.c frozen-table rule, mirrors window.c).
   STR keeps the serial append path.
2. exec_if_selected: bail to eager when both branches are trivial (column
   scan / scalar const) and eager fills the type combination correctly —
   the lazy path only pays off when a branch is an expression worth
   restricting to its passing rows.  Mixed numeric/string shapes stay on
   the selected path (its per-value string conversion).

100M rows local c24: numeric if 1883->310ms, sym if 2012->306ms.
dazzle c48 canonical Q22: 2658->1333ms end-to-end.
make test 3635/3635.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* perf(filter): parallel bitmap->index build in exec_filter and sel_compact

exec_filter ran two sequential 0..nrows sweeps (pass-count and
match_idx build) before its parallel gather; sel_compact rebuilt
match_idx from the rowsel serially.  Both now use the classic 3-phase
compaction: parallel per-chunk/per-seg counts, tiny serial prefix,
parallel fill at disjoint offsets.  Lazy/morsel-backed predicates keep
the sequential sweep.

100M rows local c24: 2-col where-select 143->21.7ms (1.4x -> 6.6x
scaling); if+where 1040->324ms.  make test 3635/3635.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* perf: parallel where builtin, gather_by_idx, and chunk-task dispatch

- ray_where_fn: 3-phase chunk compaction on the pool (was fully serial).
- gather_by_idx: fixed-width value gathers dispatched over disjoint
  output ranges (null-bit propagation stays serial - shared-word bit
  writes would race).
- exec_filter/where chunk phases now use ray_pool_dispatch_n (one task
  per chunk); ray_pool_dispatch morselizes total_elems by 1024, so
  passing chunk counts gave only ~2 tasks for 100M rows.

100M rows local c24: where 88->25ms, at-gather 80->31ms,
2-col where-select 138->20ms (6.8x).  make test 3635/3635.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(review): harden parallel paths per skeptic review

Blockers (DA binary-agg y-column):
- eligibility now requires a plain numeric/temporal y; nullable
  integer/temporal y stays on the HT path (da_accum_row's pair branch
  has no y-side sentinel machinery - nulls would accumulate as values)
- an FP y with HAS_NULLS sets da_any_nullable so nn[] is allocated and
  the emitter divides by the non-null PAIR count, not the group count

Majors:
- all new parallel gates require pool->n_workers > 0 (a -c 1 pool
  exists with 0 workers; ring fill + atomics + rc_sync were pure
  overhead, and the OP_IF eager reroute lost to the selected path
  serially - the Q22/Q25 c1 regression)
- chunked dispatch_n call sites cap chunks at 1024 = the pool's
  initial ring capacity, so the ring never grows (dispatch_n clamps
  and silently DROPS tasks if ring growth fails -> uninitialized
  prefix entries -> OOB writes)
- sel_compact seg fill switched to dispatch_n over seg-chunks
  (ray_pool_dispatch over segs gave 1 task under 8.4M rows)
- gather_by_idx parallel path guarded by ray_parallel_flag == 0
  (leaf utility, 35+ call sites; nested dispatch would corrupt the
  single-producer task ring)

Nits: stray time.h include, restored v2-gate comment,
RAY_PARALLEL_THRESHOLD symbol in pivot.c.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(group): pair-skip y-side nulls in the legacy HT binary-agg path

Unify grouped binary-aggregate (pearson/cov/scov/wsum/wavg) null handling
with the scalar reducers, the v2 engine and the DA path: a null on either
side of the (x,y) pair now voids the whole pair on the legacy HT route too.

- ght_compute_layout: a nullable y-side sets GHT_AF2_Y_NULLABLE and routes
  the layout to the null-aware accumulators.
- accum_from_entry_nullable: pair-skip before nn++/sums.
- Entry packing canonicalizes integer nulls so the accumulator can see
  them: NaN in F64-packed slots (a (double)sentinel cast previously read
  as a huge finite value — this also fixes nullable-int x beside an FP y),
  NULL_I64 in int-by-int slots.
- Both HT emitters (radix + serial) divided pearson/cov/scov moments by
  the group row count instead of the accumulated pair count — wrong
  results whenever a group carried any null; now divide by nn.
- The all-null-group guards wrote v=0.0 after ray_vec_set_null, so the
  common store overwrote the null sentinel — emit NULL_F64 instead.
- DA eligibility now also rejects a y shorter than the scan (OP_CONST
  vector literal would read out of bounds).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JZwH1gpAeLzDX2bxc4jz6s

* test(agg): cross-path null coverage for grouped binary aggregates

46 assertions for wsum/wavg/pearson_corr/cov/scov over nullable inputs on
all three grouped routes — v2 (plain-scan int key), DA (expression int
key), legacy HT (expression key + nullable-int y; F64-packed and
int-packed entry lanes) — against independently computed pair-skip truth,
for all four x/y type combinations, plus an all-pairs-null group
(wsum 0.0, typed nulls for the ratio/moment aggs) on every route.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JZwH1gpAeLzDX2bxc4jz6s

* chore(review): shared dispatch-safety gate; single filter threshold

- ops/internal.h ray_par_dispatch_ok(): workers + RAY_PARALLEL_THRESHOLD +
  ray_parallel_flag reentrancy check in one place; applied at exec_filter,
  sel_compact, exec_if_eager and the where builtin (local copy there —
  builtins.c cannot include ops/internal.h).
- exec_filter: gate and table fallback derive from one row count
  (fidx_rows); note that pass_count from the parallel count phase is
  consumed by exec_filter_vec for vector inputs.
- group.c: drop the never-read da_ctx_t.agg_pair_mask plumbing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JZwH1gpAeLzDX2bxc4jz6s

* chore(par): shared dispatch predicate, ring-cap constant, parallel-path test

Follow-ups from the audit's non-blocking notes:

- core/pool.h ray_pool_par_dispatch_ok(pool, n, min_elems): the single
  home for the dispatch-safety predicate (workers + element threshold +
  ray_parallel_flag reentrancy).  The three hand-copies in ops/internal.h,
  lang/eval.c and ops/builtins.c are gone; all six gates call the shared
  one.
- RAY_POOL_INIT_TASKS in core/pool.h replaces the hardcoded 1024 at the
  three dispatch_n chunk caps and in ray_pool_create, with a
  _Static_assert tying it to RAY_POOL_MAX_TASKS — lowering the initial
  ring capacity can no longer silently desync from the caps that rely
  on it.
- test/rfl/query/parallel_paths_large.rfl: 200k-row coverage of every
  new pool-parallel branch (where, gather-by-index, exec_filter,
  sel_compact, OP_IF numeric and SYM fill incl. the serial LUT warm-up)
  against closed-form expected values.
- RAY_F32 dropped from if_type_eager_ok's whitelist (if_fill_range has
  no F32 case; unreachable today, kept unreachable deliberately).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JZwH1gpAeLzDX2bxc4jz6s

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Anton Kundenko <singaraiona@gmail.com>

* fix(aggr): preserve slice nullability in binary groups

* fix(expr): avoid f64 null cast in fallback idiv integer output (#344)

binary_range's OP_IDIV kernels for narrow integer output (I64/I32/I16/U8)
computed `(intN_t)floor(lv/rv)` guarded only by `rv != 0.0`. That guard
does not catch a NaN operand (`NaN != 0.0` is true), so a null float input
yields `lv/rv == NaN`, `floor(NaN) == NaN`, and the subsequent cast to an
integer type is undefined behavior — UBSan: "nan is outside the range of
representable values of type 'long long'" at exec/expr_binary_f64_idiv_mod.

Route the cast through the ray_cast_f64_to_{i64,i32,i16,u8}_null helpers,
which map NaN to the canonical null sentinel (NULL_I64/I32/I16, 0 for the
non-nullable U8) and saturate out-of-range finite results. The null
post-pass (propagate_nulls_binary) already overwrites these positions, so
final values are unchanged — this only removes the UB and yields the
correct sentinel in-buffer. Mirrors the already-safe F64-output IDIV arm
(ray_f64_fin) and the sibling casts fixed in "avoid f64 null casts to
integers"; depends on those helpers.

Co-authored-by: Evgen Byelozorov <belowzeroff@gmail.com>

* fix(group): avoid f64 null read cast in dense aggs (#343)

Co-authored-by: Evgen Byelozorov <belowzeroff@gmail.com>
Co-authored-by: Anton Kundenko <singaraiona@gmail.com>

* fix(ipc): preserve boxed data list args (#346)

* fix(group): avoid f64 null cast in DA reads (#348)

* ci: use portable march for fuzz jobs (#347)

Co-authored-by: Anton Kundenko <singaraiona@gmail.com>

* fix(pivot): preserve generic missing cells as null (#350)

* fix(xbar): avoid narrow bucket truncation (#351)

* fix(arith): reject float temporal operands (#353)

* fix(ops): make nested (LIST) columns usable through a parted view (fixes #355) (#356)

* fix(query): preserve temporal arithmetic semantics (#354)

* ci: skip audit comments for cancelled runs

* fix(query): preserve if temporal branch types (#359)

* fix(store): reject duplicate splayed column names (#360)

* fix(store): reject duplicate splayed column names

* test(ci): harden Ctrl-C PTY synchronization

---------

Co-authored-by: Anton <singaraiona@gmail.com>

* fix(builtins): reject malformed strings in TIMESTAMP cast (#361)

* fix(builtins): reject malformed strings in TIMESTAMP cast

(as 'TIMESTAMP str) accepted a range of malformed inputs and silently
produced a valid-looking but wrong value — invisible data corruption at
the call site. Examples that used to succeed:

- "2024-01-02x01:02:03", "2024-01-02abc"  -> midnight (time dropped)
- "2024-01-02T25:02:03"                    -> rolled into the next day
- "2024-01-02T12:34junk", "...03Zjunk"     -> trailing garbage ignored
- "2024-01-02T12:34:"                       -> dangling component ignored
- "2024-01-02T12:34:03+99:99"               -> out-of-range tz, wrong date

Root cause: the parser used unanchored sscanf calls that matched a prefix
and ignored the rest, and it never range-checked the components.

Replace it with a bounded cursor over the grammar

    YYYY<sep>MM<sep>DD [ (T|' '|D) HH:MM[:SS][.frac] [Z|(+|-)HH[:]?MM] ]

(<sep> is '-' or '.', consistent within the date). The cursor must reach
the end of the string, every field is a fixed digit width, and the date,
time, and timezone components are range-checked; anything else is a domain
error. All previously accepted valid forms — bare date, space/T/D
separators, fractional seconds (any length), and Z / +HH:MM / -HHMM /
offset suffixes — continue to round-trip.

Extends the TIMESTAMP-cast coverage in ops/builtins_branch_cov.rfl with
the separator, out-of-range, trailing-garbage, partial-component, and
out-of-range-timezone rejection cases.

* fix(builtins): validate timestamp cast bounds

---------

Co-authored-by: Anton Kundenko <singaraiona@gmail.com>

* fix(store): support nested column inserts and persistence (#365)

* fix(eval): enforce restricted mode in compiled lambdas (#366)

* feat(core): add bounded poll step for embedders (#367)

* fix(test): rely on public runtime declarations (#369)

* feat: website add consumer

* feat(core): expose restricted poll mode (#372)

* fix(aggr): preserve grouped nested first and last

* fix(join): preserve nested columns (#376)

* fix(csv): round-trip signed and >=24h TIME values in .csv.read (#379)

* fix(str): propagate null start/length in substr instead of overflowing (#378)

* fix(builtins): clamp out-of-range float in scalar numeric casts (#380)

* fix(builtins): validate bounds in DATE string cast (#382)

(as 'DATE str) parsed "YYYY.MM.DD" with an unanchored sscanf and never
range-checked the month. A month greater than 13 walked the days-in-month
table out of bounds — e.g. (as 'DATE "9999.99.99") reads md[99] on a
13-element array, which ASan reports as a heap/global out-of-bounds read
(builtins.c:1567). The same path also silently accepted trailing garbage
("2024.01.02junk") and impossible days ("2024.02.31").

Replace it with a bounded cursor over YYYY.MM.DD that must consume the
whole string, validates the month (1-12) BEFORE indexing the table, and
validates the day against the actual number of days in that month
(leap-aware). This mirrors the strict TIMESTAMP string cast. All valid
dates — including the leap day 2024.02.29 — still round-trip; malformed
input now returns a domain error instead of reading out of bounds or
fabricating a date.

Adds DATE-cast rejection/validation coverage to type/as.rfl.

* perf(aggr): specialize reduction scans (#383)

* fix(join): preserve nested columns

Gather LIST columns with retained ownership across equi, anti, and asof joins. Represent unmatched boxed rows with the runtime null singleton and propagate allocation failures without dropping columns.\n\nFixes #375

* perf(aggr): specialize reduction scans

* perf(collection): parallel radix distinct for fixed-width columns (#384)

* perf(collection): parallel radix distinct for fixed-width columns

distinct_vec_eager's hashset pass is single-threaded and dominates on
large numeric/temporal columns. Reuse exec_count_distinct's radix
layout (histogram -> scatter -> per-partition dedup) carrying row ids
alongside values; first occurrences land in a shared byte array (a
value lives in exactly one partition, so no atomics) and feed the
existing sort + gather tail — result semantics unchanged.

13.9M-row column, 10 cores: 11.6K uniques 106->26ms, 125K uniques
164->45ms, 1.1M uniques 294->133ms, 5M uniques 719->502ms (the shared
sequential value-sort now dominates that last case).

* test(collection): cover the parallel radix distinct path

The existing distinct.rfl only exercises small vectors; the radix
kernel engages at >= 65536 rows. Add radix-scale assertions for every
lane it handles (i64, i32, i16, f64, time, timestamp): value-sorted
output, declared nulls collapsing to one sentinel, idempotence, the
dedup invariant, agreement with the fused OP_COUNT_DISTINCT kernel,
and the exact engage-threshold boundary. F64 NaN ordering among sorted
values is comparator-defined, so those asserts check cardinality and
null survival rather than full ordering.

* fix(csv): harden import cancellation and schema handling (#387)

* fix(csv): harden import cancellation and schemas

* ci(tsan): use portable x86-64 baseline

* fix(temporal): guard int64 overflow in DATE/TIMESTAMP decomposition (#386)

* fix(temporal): guard int64 overflow in DATE/TIMESTAMP decomposition

The DATE/TIMESTAMP → microseconds conversion shared by the temporal
extract and truncate paths overflowed int64 on extreme inputs (UBSan):

  * DATE is int32 days, so an extreme value × µs-per-day overflowed —
    (dd (as 'DATE 2147483647)) tripped `raw * 86400000000`.
  * The ns→µs floor for TIMESTAMP negated the input, so a value within
    999 of INT64_MIN overflowed `(-raw) + 999`.
  * The DAG date_trunc YEAR/MONTH arms re-multiply days_from_civil(...) —
    a day count floored down to the period start, up to a year beyond the
    input — by µs-per-day, so a DATE that cleared the µs bound still
    overflowed (d.year of (as 'DATE -106751991)).

Both conversions appear in all four decomposition kernels: the standalone
ray_temporal_extract / ray_temporal_truncate and the DAG exec_extract /
exec_date_trunc morsel kernels.

Do the TIMESTAMP ns→µs floor with truncate-then-adjust so it never
negates (overflow-free, exact at INT64_MIN). A DATE so extreme its µs
value is not representable decodes to a null instead of reading overflow
garbage, consistent with how these kernels already treat a null input.
The truncate DATE bound is the int64-NANOSECOND representable day range
(not just the µs one) so the YEAR/MONTH re-multiply cannot overflow, and
truncate additionally nulls a result whose bucketed µs would overflow the
ns output.

Adds temporal/extract_trunc_overflow.rfl covering the standalone kernels
(yyyy/dd/mm/hh and (date …)) and the DAG kernels (dotted col.field),
including the year/month re-multiply path, large-magnitude negative DATE,
and minimum-edge TIMESTAMP.

* fix(temporal): range-check truncated result, not pre-floor headroom

rte_trunc_elem rejected a truncation whenever the input fell within one
bucket of the low int64-ns edge, even when the floored result was still
representable — so (date (as 'TIMESTAMP -9223286400000000000)) returned 0Np
though its day boundary 1707.09.23 is a valid TIMESTAMP that the DAG path
(select ts.date) returned, leaving the two public truncate paths disagreeing
(PR #386 review).

Floor with overflow-safe arithmetic (truncate toward zero, then guard the
single toward-minus-infinity bucket subtraction) and range-check the actual
bucketed result against the ns domain, mirroring exec_date_trunc. Adds the
low-boundary case — standalone and DAG — as a regression.

---------

Co-authored-by: Anton Kundenko <singaraiona@gmail.com>

* fix(eval): materialize lazy values at compiled boundaries (#391)

* fix(arith): wrap scalar integer add/sub/mul on overflow instead of UB (#388)

ray_add_fn / ray_sub_fn / ray_mul_fn computed `as_i64(a) OP as_i64(b)`
directly, so an i64 result that overflowed was signed-integer-overflow
undefined behavior. UBSan reported it at src/ops/arith.c:141 (+), :223
(-), and :257 (*) for e.g. (+ 9223372036854775807 1), (- 0
-9223372036854775808), and (* 9223372036854775807 2).

Compute the integer result with uint64 wraparound, matching the vector
arithmetic kernel (src/ops/expr.c OP_ADD/SUB/MUL). The wrapped value —
INT64_MIN for MAX+1, which is the i64 null sentinel — is exactly what the
code already returned by relying on the overflow, so results are
unchanged; only the undefined arithmetic is removed. Narrow-type (i16/i32)
results are unaffected: their i64 intermediate never overflows and
make_typed_int still narrows them.

Adds test/rfl/arith/overflow_wrap.rfl.

Co-authored-by: Anton Kundenko <singaraiona@gmail.com>

* fix(builtins): reject malformed strings in TIME cast (#389)

(as 'TIME str) parsed with an unanchored sscanf and never range-checked
the minute/second fields, so it silently accepted malformed input and
produced a wrong-but-valid value — the same data-corruption class the
DATE (#382) and TIMESTAMP (#361) string casts were hardened against, but
the TIME cast was left on the old path:

- "12:34:56junk"  -> 12:34:56.000  (trailing garbage ignored)
- "25:99:99"      -> 26:40:39.000  (minute/second out of range, normalized)

Parse "[-]HH:MM[:SS][.fff]" with a bounded cursor that must consume the
whole string, validating that minutes and seconds are 0-59. TIME is a
signed duration — its ms-of-day may exceed a day and go negative (see
.csv.read round-tripping) — so the hour field stays variable width and
unbounded, and "HH:MM" without seconds plus a bare trailing "." are still
accepted. The value is range-checked against the int32 millisecond domain.
The string-vector cast routes through this same atom path, so it is fixed
too.

Extends the TIME-cast coverage in type/as.rfl with the duration, negative,
and rejection cases.

Co-authored-by: Anton Kundenko <singaraiona@gmail.com>

---------

Co-authored-by: Karim <k.nassar@lynxtrading.com>
Co-authored-by: Evgen <ebelozerov@lynxtrading.com>
Co-authored-by: Evgen Byelozorov <belowzeroff@gmail.com>
Co-authored-by: Evgen Belozerov <yevhenbielozorov@gmail.com>
Co-authored-by: Serhii Savchuk <ser.vasilich@hotmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants